So besides supplying power to the system unit, the power supply serves another function: cooling your PC. Your PC is full of electronic components. The main characteristic of any electronic component is it produces heat. Over a long period of time, this heat can have a damaging effect on components if it is not removed. The cooling fan in the power supply circulates air and prevents you PC from overheating.
When you turn your PC on you will notice a slight humming sound. The spinning of the cooling fan in the power supply is the noise you hear.
Finally the power supply plays one other important role: it will prevent your PC from starting if there is not sufficient to run your PC properly.